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The  purpose  of  this  effort  was  to  develop  an  intelligent  tutoring  system  (ITS)  to  train  test  administrators  how  to 
operate  computerized  adaptive  testing  Armed  Services  Vocational  Aptitude  Battery  (CAT-ASVAB).  Once 
developed,  a  final  troubleshooting  lesson  was  completed  to  (a)  test  the  contributions  of  coaching  and  information 
processing  strategies  on  ITS  training  effectiveness,  and  (b)  develop  a  general  personal  computer  (PC) 
Troubleshooting  ITS  for  use  Navy-wide.  The  ITS  uses  information-processing  efficiency  data  along  with 
measures  of  correctness  in  determining  the  extent  to  which  students  have  successfully  compiled  information  and 
what  training  to  present  next. 

We  hypothesized  that  information-processing  training  improves  retention  of  skills  in  an  ITS.  Specifically,  we 
hypothesized  that  examinees  who  have  the  information-processing  features  in  their  ITS  compile  the 
troubleshooting  declarative  and  procedural  knowledge  more  completely  so  that  they  have  less  decrement  in 
troubleshooting  tasks  over  time. 


Method 


Participants 

The  sample  was  comprised  of  767  paid  participants,  between  17-35  years  old.  The  sample  was  assembled  to 
resemble  current  or  future  field  test  administrators  (TAs)  with  respect  to  education  level,  age,  primary  language, 
citizenship,  and  absence  of  diagnosed  reading  difficulties. 

Instruments:  Description  of  the  Research  ITS 

The  research  version  of  the  ITS  contains  two  lessons  and  two  final  exams.  The  first  lesson,  Introduction  to 
Computers  and  the  CAT-ASVAB  System,  presents  conceptual  and  factual  information  about  computers  and  the 
CAT-ASVAB  testing  system  using  a  tutorial  strategy.  The  ITS  includes  topic,  test  question,  remediation,  and 
performance  feedback  screens. 

Lesson  One,  Introduction  to  Computers  and  the  CAT-ASVAB  System 

Lesson  content.  Lesson  1  contains  instruction  in  three  topics:  (1)  CAT-ASVAB  Hardware,  (2)  CAT-ASVAB 
Test  Stations,  and  (3)  Test  Modes.  CAT-ASVAB  hardware  includes:  a  monitor,  a  DOS-based  computer,  a 
keyboard  and  the  associated  cables.  The  CAT-ASVAB  also  includes  three  different  types  of  test  stations:  (1)  the 
Test  Administrator  (TA)  Station,  (2)  the  Examinee  Testing  (ET)  Station,  and  (3)  the  File  Server.  The  Test 
Administrator  uses  the  TA  Station  to  help  with  administering  CAT-ASVAB  and  to  manage  the  test  sessions. 
Examinees  use  the  ET  Stations  to  take  the  test,  and  the  File  Server  allows  for  communication  between  the  TA 
and  ET  Stations. 

Instructional  design.  Lesson  One  uses  simple,  algorithmic  branching  techniques  to  present  conceptual  and  factual 
information  in  a  tutorial  strategy.  For  each  lesson  topic,  the  program  presents  a  series  of  instruction  screens,  a 
series  of  review  screens,  a  topic  quiz,  remediation  screens  if  the  student  answers  quiz  items  incorrectly,  and 
performance  feedback.  The  screens  are  presented  in  a  simple  linear  design. 

Performance  assessment.  A  final  exam  covering  all  of  the  content  in  Lesson  One  appears  at  the  end  of  Lesson 
One.  Immediate  feedback  is  presented  after  each  test  item,  and  an  overall  feedback  screen  appears  at  the  end  of 
the  final  exam. 


Lesson  Two.  Declarative  Knowledge 
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Lesson  content.  The  second  lesson,  CAT-ASVAB  Troubleshooting,  is  divided  into  two  sections:  (1)  declarative 
knowledge  and  (2)  troubleshooting  scenarios.  The  declarative  knowledge  section  teaches  fundamental 
CAT-ASVAB  troubleshooting  principles  and  methods  and  obtains  an  initial  estimate  of  students’  troubleshooting 
abilities  on  four  topic:  (1)  the  definition  of  troubleshooting,  (2)  preventive  maintenance,  (3)  PC  troubleshooting 
rules,  and  (4)  introduction  to  scenarios. 

Instructional  design.  The  design  of  this  first  section  of  Lesson  Two  is  similar  to  that  of  Lesson  One.  Instruction, 
review,  feedback,  and  remediation  screens  are  included. 

Performance  assessment.  The  performance  assessment  for  this  part  of  Lesson  2  contains  multiple-choice 
trouble-shooting  problems  with  immediate  feedback  and  remediation,  as  necessary.  The  problems  build  on  the 
knowledge  that  the  students  are  acquiring  as  they  progress  through  the  lesson.  Student  performance  on  the  first 
section  of  Lesson  Two  provides  input  for  use  by  the  program  in  the  next  section  of  Lesson  Two.  Every  question 
appearing  in  the  declarative  knowledge  section  is  associated  with  one  or  more  of  the  30  problem-solving  rules 
that  make  up  the  domain  expert  for  the  trouble-shooting  scenarios. 

Lesson  Two.  Troubleshooting  Scenarios 

The  troubleshooting  principles  and  methods  learned  in  the  declarative  knowledge  section  of  the  lesson  are 
applied.  This  section  uses  artificial  intelligence  components  to  develop  a  model  of  student  performance  and 
compare  that  model  with  the  program’s  expert  model. 

User  interface.  Every  scenario  has  three  sections:  Introduction;  Isolate  Problem;  and  Review.  The  Introduction 
section  describes  what  took  place  before  the  CAT-ASVAB  system  failure  occurred  and  provides  the  student  with 
the  needed  background  information  to  effectively  troubleshoot  the  failure.  The  Introduction  part  of  the  scenarios 
consists  of  four  to  five  screens.  The  Isolate  Problem  section  provides  the  student  with  nine  different  menus  from 
which  to  choose  a  troubleshooting  procedure.  The  Isolate  Problem  section  provides  students  with  two  additional 
menu  options:  DIAGRAM  and  STEPS.  When  students  select  the  DIAGRAM  option,  it  displays  a  graphic 
depiction  of  the  test  station(s)  in  which  the  failure  is  occurring.  When  a  student  selects  the  STEPS  menu  option, 
it  presents  the  troubleshooting  steps  the  student  already  has  performed.  All  troubleshooting  scenarios  end  with  a 
review,  consisting  of  one  to  nine  screens,  that  summarizes  the  troubleshooting  steps  taken  to  correct  the  failure. 

Lesson  content.  We  performed  a  cognitive  task  analysis  to  derive  74  scenarios  that  the  program  uses  for 
instructional  and  practice  purposes.  These  scenarios  require  students  to  correctly  select  and  apply  one  or  more  of 
the  30  problem-solving  rules  in  the  domain  expert.  In  addition,  we  developed  an  additional  20  scenarios  that  the 
program  uses  only  for  performance  assessment. 

Instructional  design  and  performance  assessment.  The  highly  modular  software  contains  a  domain  expert,  student 
model,  a  diagnostic  module,  and  a  tutorial  module.  The  domain  expert,  or  expert  model,  contains  the 
problem-solving  rules  that  the  student  needs  to  learn.  The  student  model  maintains  a  current  profile  of  what  the 
student  does  and  does  not  know  based  upon  the  available  information  and  treatment  condition.  The  diagnostic 
module  uses  a  Bayesian  inferencing  approach  to  assess  the  probability  that  the  student  would  correctly  apply  a 
given  problem-solving  rule,  and  the  tutorial  module  selects  the  next  troubleshooting  scenario  based  on  the 
diagnostic  module’s  assessment  of  the  current  state  of  the  student  model. 

The  second  part  of  Lesson  Two  consists  entirely  of  troubleshooting  scenarios.  It  requires  students  to  apply 
troubleshooting  principles  and  methods.  A  troubleshooting  scenario  begins  with  a  series  of  screens  that  describe 
a  CAT-ASVAB  system  failure  and  then  provides  response  options  for  students  to  correct  the  failure.  As  students 
complete  the  scenarios,  the  program  updates  its  model  of  the  student’s  current  knowledge  state,  and  the 
program’s  diagnostic  module  uses  artificial  intelligence  features  to  compare  the  updated  student  model  with  the 
expert  model.  Based  on  that  information,  the  program’s  tutorial  module  selects  the  best  teaching  strategy  and 
scenario  to  present.  The  course  is  self-paced. 

Procedure:  Treatment  Conditions 


For  research  purposes,  we  developed  for  different  versions  of  the  troubleshooting  scenarios.  The  four  versions 
are  (1)  baseline,  (2)  coaching,  (3)  information  processing,  and  (4)  coaching  plus  information  processing. 

Baseline.  The  control  condition  includes  the  same  Bayesian  inferencing  approach  and  performance  assessment 
components  as  the  other  conditions.  Corrective  feedback  explains  why  the  student’s  choice  is  an  error  and 
indicates  that  there  is  a  better  approach  to  a  given  step  without  directly  suggesting  a  repair.  After  three  incorrect 
answers  to  the  same  step  the  baseline  condition  provides  the  correct  answer.  The  Baseline  condition  does  not 
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include  any  of  the  additional  experimental  features. 

Coaching.  The  coaching  condition  adds  coaching  components  to  the  baseline  version  (see  Graesser,  Person,  & 
Huber,  1995;  Shaw,  Dunkle,  &  Bendixen,  1995).  These  coaching  components  include  a  student-controlled  HINT 
menu  option  as  well  as  a  program-controlled  coach.  When  selected  by  the  student,  the  HINT  menu  option 
displays  advice  about  which  problem-solving  step  the  student  should  take  next.  To  avoid  students  overusing 
hints,  program  instructions  inform  students  that  their  progress  through  training  depends  on  eventually  solving 
problems  without  the  aid  of  the  HINT  key.  Therefore,  the  program  updates  the  student  model  based  on  whether 
or  not  the  student  accesses  the  HINT  key.  In  addition,  the  tutor  fills  in  when  the  student  cannot  perform  all  the 
steps  in  a  skill,  but  it  facilitates  the  student’s  providing  more  and  more  of  the  work  until  the  tutor  operates 
completely  in  the  background. 

Information  Processing.  Littman  and  Soloway  (1988)  stress  the  importance  of  including  process  measures  of 
performance  when  assessing  the  effectiveness  on  an  ITS.  They  note  that  the  purpose  of  the  student  or  diagnostic 
model  is  to  capture  how  students  solve  problems  and  not  merely  that  they  can  solve  particular  problems.  We 
developed  measures  of  information-processing  efficiency  that  reflect  what  the  student  knows  and  how  the 
student  evaluates  and  solves  cognitively  complex  tasks  (see  Dillon,  1986,  1997).  The  information-processing 
condition  adds  information-processing  treatment  and  measurement  components  to  the  baseline  version.  To 
implement  this  condition,  we  added  features  to  the  interface,  developed  and  tracked  additional 
information-processing  measures,  and  added  information-processing  efficiency  performance  criteria  to  the 
baseline  ITS. 

Information-processing  interface.  We  added  user-controlled,  selective  disclosure  windows  to  track  students’ 
information-processing  strategies.  For  the  information-processing  condition,  the  declarative  section  of  Lesson 
Two  provides  instruction  on  the  most  efficient  way  to  troubleshoot  failures  using  the  selective-disclosure 
windows.  That  lesson  instructs  students  to  use  their  time  efficiently  by  viewing  the  information  available, 
thinking  about  the  six  troubleshooting  rules,  and  thinking  about  how  to  apply  them  to  the  hardware  failure.  So, 
the  information  in  the  lesson  was  taught  using  instruction  in  learning  material  by  processing  material  efficiently 

Information-processing  measures.  The  program  collects  different  kinds  of  data  regarding  how  students  process 
information  within  a  particular  window  specifically  (1)  the  number  of  times  a  student  views  the  menu  window 
(Menu  Views),  and  (2)  the  amount  of  time  spent  with  the  menu  covered  (Diagnostic  Time),  and  (3)  with  the 
menu  uncovered  (Procedure  Selection  Time).  Using  these  data,  the  program  computes  Information-  Processing 
Efficiency  (IPE)  scores  and  Information-Processing  Learning  (IPL)  scores.  The  program  also  computes  another 
IPE  score  based  on  the  time  data.  The  program  computes  the  Proportion  of  Diagnostic  Time  (PDT)  as  a  ratio  of 
Diagnostic  Time  to  total  problem  time:  PDT  =  (Diagnostic  Time)  /  (Procedure  Selection  Time  +  Diagnostic 
Time).  As  explained  in  the  declarative  part  of  Lesson  Two,  students  can  perform  most  efficiently  in  they  spend 
more  of  their  time  in  Window  1  (i.e.,  with  the  window  covered)  getting  all  the  information  they  need  (their 
Diagnostic  Time)  and  then  selecting  and  confirming  their  answer  (Procedure  Selection  Time).  Finally,  the 
program  calculates  two  variables  that  assess  students’  improvement  over  time  on  the  IPE  variables.  These 
variables  are  the  Information  Processing  Learning  (IPL)  scores.  Learning  indices  measure  changes;  i.e., 
improvements  in  the  above  information-processing  measures. 

Information-processing  efficiency  criteria  and  feedback.  The  information-processing  condition  attempts  to 
improve  students’  performance  both  by  improving  their  information-processing  efficiency  and  by  insuring  that 
they  process  information  efficiently  before  allowing  them  to  progress  in  the  tutor.  Information-processing 
efficiency  results  in  information-processing  feedback  in  addition  to  the  regular  feedback  that  the  student 
receives,  such  as  the  following  statements:  "You  are  working  through  the  scenarios  very  efficiently!  Thinking  the 
problem  through  BEFORE  accessing  Window  #2  is  a  good  strategy."  If  students  fail  to  meet  any  of  the  efficiency 
criteria,  they  receive  corrective  information-processing  feedback  in  addition  to  the  regular  feedback. 

Coaching  Plus  Information  Processing.  Both  coaching  components  and  information-processing  components  are 
added  to  the  baseline  condition.  When  students  do  not  pass  any  of  the  information-processing  efficiency  criteria 
for  a  particular  problem-solving  rule,  the  program  displays  the  coach  for  that  problem-solving  rule  just  prior  to 
the  student’s  next  attempt  at  a  troubleshooting  task  involving  the  rule. 

Procedure:  Final  Exams  of  the  Research  ITS 


All  participants  completed  the  same  final  exams.  Test  1  was  taken  upon  completion  of  Lesson  Two.  Test  2  was 
completed  one  week  after  Test  1 .  The  exams  were  designed  to  be  as  similar  as  possible  to  the  baseline 
troubleshooting  scenarios  of  Lesson  Two.  Each  of  the  two  final  exams  contains  92  items.  We  designed  the  exams 
to  have  at  least  one  item  that  presented  an  opportunity  for  students  to  correctly  apply  each  of  the  30  different 
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problem-solving  rules  in  the  domain  expert  of  the  ITS.  In  order  to  develop  an  exam  that  covered  all  the  content 
areas,  it  was  necessary  to  duplicate  some  early  aspects  of  the  scenarios  that  lead  up  to  the  new  part  of  the 
problem.  This  duplication  meant  that,  for  some  scenarios,  students  already  had  seen  the  early  part  of  the  scenario 
in  a  previous  scenario,  and  they  already  had  received  feedback  about  the  correct  next  step.  Therefore,  we  divided 
the  finals  into  two  conceptual  parts,  one  having  "first-exposure"  items  and  the  other  having  "second-exposure" 
items.  Of  the  92  final  exam  items,  53  were  first-exposure  items,  and  39  were  second-exposure  items.  Differences 
found  between  the  four  groups  on  the  53  first-exposure  items  are  due  to  the  impact  of  the  training  alone  and  not 
to  feedback  during  the  final  exam  process. 

Results 

We  computed  alpha  reliability  coefficients  for  each  of  the  three  sets  for  the  two  tests:  all  92  items,  first-exposure 
items  only,  and  second-exposure  items  only.  Only  second  exposure  items  were  below  .80.  The  alpha  reliability  is 
.65  for  Test  1  second-exposure  items  and  .62  for  Test  2  second-exposure  items.  Test-retest  reliability  for 
second-exposure  items  is  .56.  Table  1  contains  means  and  standard  deviations  for  Test  1  and  Test  2  by  group. 


Table  1.  Means  anH  Standard  Deviations  for  Test  1  and  Test2  by  Group 


GROUP 

TEST  1 

TEST  2 

'  SD 

M 

SD 

M 

Baseline 

75.73 

11.63 

73.39 

10.50 

Coaching 

77.29 

9.35 

75.00 

9.41 

IP 

75.92 

11.52 

75.07 

11.29 

CIP 

77.13 

10.22 

75.56 

9.80 

TOTAL  76.53 

10.72 

74.74  10.29 

Note:  Each  group  had  189  participants  (total  n  =  756). 


No  differences  were  found  in  Test  1  performance  as  a  function  of  condition.  Data  reveal  greater  training  time  for 
students  who  receive  either  coaching  (F=22.41,  p  <  .05)  or  IP  (F=247.82,  p  <  .05). 

Data  also  were  analyzed  to  test  the  premise  that  information-processing  principles  improve  retention  of  training 
performance  relative  to  baseline  or  coaching  conditions.  We  are  interested  here  in  the  extent  to  which  these 
procedures  reduce  any  decrement  in  troubleshooting  skills  over  time.  Conditions  with  information-processing 
training  were  expected  to  yield  less  decrement  in  troubleshooting  from  Test  1  to  Test  2.  Students  demonstrate  a 
decrement  in  troubleshooting  performance  over  time  (F=58.89,  p  <  .05)  and  data  reveal  an  interaction  for  Time  x 
IP  (F=6.10,  p  <  .05).  Thus,  the  four  groups  taken  as  a  whole  scored,  on  average,  lower  (about  1.79  points)  when 
comparing  their  test  scores  from  Test  1  to  Test  2.  In  addition,  the  interaction  between  Time  and  IP  indicates  that 
the  difference  between  participants’  test  scores  between  Test  1  and  Test  2  depends  on  whether  or  not  they  were 
received  information-processing  training.  While  students  in  the  information-processing  conditions  do  not 
demonstrate  better  performance  during  the  final  exam  than  students  in  the  other  conditions,  they  demonstrate  less 
decrement  in  training  performance  at  Week  2. 

Discussion 

The  coaching  conditions  did  not  produce  an  effect  on  test  performance,  at  Test  1  or  Test  2.  Participants  under 
coaching  showed  a  decrement  in  test  performance  over  time.  The  decrement  in  test  performance,  however,  was 
not  different  from  the  decrement  experienced  by  students  who  did  not  receive  coaching.  And,  students  in  the 
coaching  conditions  required  more  time  (about  1 1  minutes;  .34  more  standard  deviation  units)  to  complete 
training  than  students  in  the  baseline  condition. 

Data  show  that  participants  who  receive  information-processing  training  demonstrate  a  smaller  performance 
decrement  over  time  than  students  who  do  not  receive  information-processing  procedures.  Students  in  the  IP 
condition  lost,  on  average,  only  0.85  points  (or  0.075  standard  deviation  units)  from  Test  1  to  Test  2,  while 
participants  in  the  Baseline  condition  lost  2.34  points  (or  0.21 1  standard  deviation  units).  The  data  offer 
interesting  support  for  the  premise  that  information-processing  procedures  help  students  compile  declarative 
knowledge.  Also  interestingly,  the  effect  of  information-processing  training  is  seen  one  week  after  framing  rather 
than  immediately  after  training.  Because  of  the  fundamental  nature  of  information-processing  abilities,  such 
componential  knowledge  is  well  learned  or  consolidated  over  time  and  is  resistant  to  decay. 
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Future  research  should  focus  on  assessing  students’  information-processing  abilities  prior  to  assigning  them  to 
training  conditions.  Students  high  on  information-processing  efficiency  should  be  able  more  easily  to  compile 
declarative  knowledge  into  procedural  knowledge  or  more  completely  to  strengthen  their  knowledge  and  skill  in 
memory.  Research  efforts  should  be  directed  toward  determining  what  instruction  or  which  training  conditions 
will  help  students  improve  their  information-processing  efficiency,  thereby  facilitating  the  retention  of 
cognitively  complex  knowledge  and  skills. 
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